05. Experiment Size

After computing the number of observations needed for an experiment to reliably
detect a specified level of experimental effect (i.e. statistical power), we
need to divide by the expected number of observations per day in order to get
a minimum experiment length. We want to make sure that an experiment
can be completed in a reasonable time frame so that if we do have a successful
effect, it can be deployed as soon as possible and resources can be
freed up to run new experiments. What a 'reasonable time frame' means will
depend on how important a change will be, but if the length of time is beyond a month
or two, that's probably a sign that it's too long.

There are a few ways that an experiment's duration can be reduced. We could, of
course, change our statistical parameters. Accepting higher Type I or Type II
error rates will reduce the number of observations needed. So too will
increasing the effect size: it's much easier to detect larger changes.

Another option is to change the unit of diversion. A 'wider' unit of diversion
will result in more observations being generated. For example, you could
consider moving from a cookie-based diversion in a web-based experiment to an
event-based diversion like pageviews. The tradeoff is that event-based
diversion could create inconsistent website experiences for users who visit
the site multiple times.